Most task-oriented dialogue (TOD) benchmarks assume users that know exactly how to use the system by constraining the user behaviors within the system's capabilities via strict user goals, namely "user familiarity" bias. This data bias deepens when it combines with data-driven TOD systems, as it is impossible to fathom the effect of it with existing static evaluations. Hence, we conduct an interactive user study to unveil how vulnerable TOD systems are against realistic scenarios. In particular, we compare users with 1) detailed goal instructions that conform to the system boundaries (closed-goal) and 2) vague goal instructions that are often unsupported but realistic (open-goal). Our study reveals that conversations in open-goal settings lead to catastrophic failures of the system, in which 92% of the dialogues had significant issues. Moreover, we conduct a thorough analysis to identify distinctive features between the two settings through error annotation. From this, we discover a novel "pretending" behavior, in which the system pretends to handle the user requests even though they are beyond the system's capabilities. We discuss its characteristics and toxicity while emphasizing transparency and a fallback strategy for robust TOD systems.

What if… you could join an organization that creates, resources, and builds life sciences companies that invent breakthrough technologies in order to transform health care and sustainability? Montai Health is a privately held, early-stage biotechnology company developing a platform for understanding and leveraging complex molecular interactions within organisms to solve global challenges in human health and sustainability. We are looking for an Infrastructure Engineer to join our small and growing team of machine learning scientists & engineers. Flagship Pioneering conceives, creates, resources, and develops first-in-category life sciences companies to transform human health and sustainability. Since its launch in 2000, the firm has, through its Flagship Labs unit, applied its unique hypothesis-driven innovation process to originate and foster more than 100 scientific ventures, resulting in over $50B in aggregate value.

As a project manager, you've probably engaged in a number of IT projects throughout your career, spanning complex monolithic structures to SaaS web apps. However, with the advancement of artificial intelligence and machine learning, new projects with different requirements and problems are coming onto the horizon at a rapid speed. With the rise of these technologies, it is becoming less of a "nice to have" and instead essential for technical project managers to have a healthy relationship with these concepts. According to Gartner, by 2020, AI will generate 2.3 million jobs, exceeding the 1.8 million that it will remove--generating $2.9 trillion in business value by 2021. Google's CEO goes so far as to say that "AI is one of the most important things humanity is working on. It is more profound than […] electricity or fire." With applications of artificial intelligence already disrupting industries ranging from finance to healthcare, technical PMs who can grasp this opportunity must understand how AI project management is distinct and how they can best prepare for the changing landscape. Before going deeper, it's important to have a solid understanding of what AI really is. With many different terms often used interchangeably, let's dive into the most common definitions first.

What do machine learning practitioners actually do? Written: 12 Jul 2018 by Rachel Thomas This post is part 1 of a series. Part 2 will explain AutoML and neural architecture search, and Part 3 will look at Google's AutoML in particular. There are frequent media headlines about both the scarcity of machine learning talent (see here, here, and here) and about the promises of companies claiming their products automate machine learning and eliminate the need for ML expertise altogether (see here, here, and here). In his keynote at the TensorFlow DevSummit, Google's head of AI Jeff Dean estimated that there are tens of millions of organizations that have electronic data that could be used for machine learning but lack the necessary expertise and skills.

At Search Party we are in the business of creating intelligent recruitment software. One of the problems we deal with is matching candidates and vacancies in order to create a recommendation engine. This usually requires parsing, interpreting and normalising messy, semi-/unstructured, textual data from résumés and vacancies, which is where the following come in: conditional random fields, bag-of-words, TF-IDFs, WordNet, statistical analysis, but also a lot of manual work done by linguists and domain experts for the creation of synonym lists, skill taxonomies, job title hierarchies, knowledge bases or ontologies. While these concepts are valuable for the problem we try to solve, they also require a certain amount of manual feature engineering and human expertise. This expertise is certainly a factor that makes these techniques valuable, but the question remains whether more automated approaches can be used to extract knowledge about the job space to complement these more traditional approaches.

At Search Party we are in the business of creating intelligent recruitment software. One of the problems we deal with is matching candidates and vacancies in order to create a recommendation engine. This usually requires parsing, interpreting and normalising messy, semi-/unstructured, textual data from résumés and vacancies, which is where the following come in: conditional random fields, bag-of-words, TF-IDFs, WordNet, statistical analysis, but also a lot of manual work done by linguists and domain experts for the creation of synonym lists, skill taxonomies, job title hierarchies, knowledge bases or ontologies. While these concepts are valuable for the problem we try to solve, they also require a certain amount of manual feature engineering and human expertise. This expertise is certainly a factor that makes these techniques valuable, but the question remains whether more automated approaches can be used to extract knowledge about the job space to complement these more traditional approaches.